Real-time image resolution measurement for single molecule localization microscopy.

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy.

Single molecule localization microscopy (SMLM) is a mainstream method in the field of super-resolution fluorescence microscopy that can achieve a spatial resolution of 20∼30 nm through a simple optical system. SMLM usually requires thousands of raw images to reconstruct a super-resolution image, and thus suffers from a slow imaging speed. Recently, several methods based on image inpainting have been developed to enhance the imaging speed of SMLM. However, these image inpainting methods may also produce erroneous local features (or called image artifacts), for example, incorrectly joined or split filaments. In this study, we use the ResNet generator, a network with strong local feature extraction capability, to replace the popularly-used U-Net generator to minimize the image artifact problem in current image inpainting methods, and develop an image inpainting method called DI-STORM. We validate our method using both simulated and experimental data, and demonstrate that DI-STORM has the best acceleration capability and produces the least artifacts in the repaired images, as compared with VDSR (the simplest CNN-based image inpainting method in SMLM) and ANNA-PALM (the best GAN-based image inpainting method in SMLM). We believe that DI-STORM could facilitate the application of deep learning-based image inpainting methods for SMLM.

Pushing the colorimetry camera-based fluorescence microscopy to low light imaging by denoising and dye combination.

Colorimetry camera-based fluorescence microscopy (CCFM) is a single-frame imaging method for observing multiple biological events simultaneously. Compared with the traditional multi-color fluorescence microscopy methods based on sequential excitation or spectral splitting, the CCFM method simplifies multi-color fluorescence imaging experiments, while keeping a high spatial resolution. However, when the level of the detected fluorescence signal decreases, the image quality, the demosaicking algorithm precision, and the discrimination of fluorescence channels on the colorimetry camera will also decrease. Thus, CCFM has a poor color resolution under a low signal level. For example, the crosstalk will be higher than 10% when the signal is less than 100 photons/pixel. To solve this problem, we developed a new algorithm that combines sCMOS noise correction with demosaicking, and a dye selection method based on the spectral response characteristics of the colorimetry camera. By combining the above two strategies, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and low light single-frame four-color fluorescence imaging was successfully performed on fixed cos-7 cells. This study expands the power of the CCFM method, and provides a simple and efficient way for various bioimaging applications in low-light conditions.

LuckyProfiler: an ImageJ plug-in capable of quantifying FWHM resolution easily and effectively for super-resolution images.

Quantifying the resolution of a super-resolution image is vital for biologists trying to apply super-resolution microscopy in various research fields. Among the reported image resolution estimation methods, the one that calculates the full width at half maximum (FWHM) of line profile, called FWHM resolution, continues the traditional resolution criteria and has been popularly used by many researchers. However, quantifying the FWHM resolution of a super-resolution image is a time-consuming, labor-intensive, and error-prone process because this method typically involves a manual and careful selection of one or several of the smallest structures. In this paper, we investigate the influencing factors in FWHM resolution quantification systematically and present an ImageJ plug-in called LuckyProfiler for biologists so that they can have an easy and effective way of quantifying the FWHM resolution of super-resolution images.

PCIe-based FPGA-GPU heterogeneous computation for real-time multi-emitter fitting in super-resolution localization microscopy.

Real-time multi-emitter fitting is a key technology for advancing super-resolution localization microscopy (SRLM), especially when it is necessary to achieve dynamic imaging quality control and/or optimization of experimental conditions. However, with the increase of activation densities, the requirements in the computing resources would increase rapidly due to the complexity of the fitting algorithms, making it difficult to realize real-time multi-emitter fitting for emitter density more than 0.6 mol/µm2 in large field of view (FOV), even after acceleration with the popular Graphics Processing Unit (GPU) computation. Here we adopt the task parallelism strategy in computer science to construct a Peripheral Component Interconnect Express (PCIe) based all-in-one heterogeneous computing platform (AIO-HCP), where the data between two major parallel computing hardware, Field Programmable Gate Array (FPGA) and GPU, are interacted directly and executed simultaneously. Using simulated and experimental data, we verify that AIO-HCP could achieve a data throughput of up to ∼ 1.561 GB/s between FPGA and GPU. With this new platform, we develop a multi-emitter fitting method, called AIO-STORM, under big data stream parallel scheduling. We show that AIO-STORM is capable of providing real-time image processing on raw images with 100 µm × 100 µm FOV, 10 ms exposure time and 5.5 mol/µm2 structure density, without scarifying image quality. This study overcomes the data throughput limitation of heterogeneous devices, demonstrates the power of the PCIe-based heterogeneous computation platform, and offers opportunities for multi-scale stitching of super-resolution images.

Single-shot multi-color fluorescence microscopy via a colorimetry camera.

Multi-color fluorescence microscopy presents highly detailed biological samples interactively. However, current multi-color methods suffer from an intricate optical setup, complicated image analysis, or a long acquisition time. To address these issues, here we develop a simple multi-color method based on a customized colorimetry camera to enable the detection of multiple structures from single-shot acquisition. The unfiltered channel (W pixels) and color channels (R, G, B, and NIR pixels) in this customized camera simultaneously provide a broad detection wavelength range and high detection sensitivity. We built a simple optical setup by replacing the monochrome camera in a basic fluorescence microscopy system with a colorimetry camera, and developed effective image analysis procedures to reconstruct a multi-color image from a single frame of a raw image. We demonstrated single-shot four-color wide-field fluorescence imaging on fixed cos-7 cells with < 5% cross talk, which is comparable to the best reported values. Our method greatly simplifies both the optical system and image analysis in the widely used method of multi-color fluorescence microscopy, thus offering an effective and easy way to study multiple objects at the same time.

A labeling strategy with effective preservation of fluorophores for expansion single-molecule localization microscopy (Ex-SMLM).

Expansion microscopy (ExM) significantly improves the resolution of conventional diffraction-limited optical microscopy by using physically expanding biological samples. Combining ExM with single-molecule localization microscopy (SMLM) could further enhance the resolving power of SMLM, which is typically in the order of 20-30 nm. However, to make this combination successful, we need to solve three key issues related to sample preparation, including mainly hydrogel shrinking in an ionic photoswitching buffer, fluorescence photobleaching due to a free-radical reaction and reduced labelling efficiency from protease digestion. Re-embedding polyacrylamide gel or using an improved photoswitching buffer with a low ionic strength is able to minimize or even solve the hydrogel shrinking problem, while the development of post-expansion labelling approaches avoids fluorescence bleaching. However, the preservation of protein epitopes (which determines the labelling efficiency) remains to be challenging. In this paper, we propose to tackle this challenge by introducing the highly selective and stable biotin-streptavidin interaction into the post-expansion labelling strategy. After upgrading the popular immunolabelling linkage scheme from Epitope-Primary antibody-Secondary antibody-Fluorophores to Epitope-Primary antibody-Secondary antibody-Biotin-Streptavidin-Fluorophores, we were able to label protein epitopes with biotin, which was stable during the expansion process, and thus avoid the troublesome problem in preserving protein epitopes or antibodies. We demonstrate that combining Ex-SMLM with the new post-expansion linkage scheme enables new possibilities in resolving the detailed arrangement of Nup133 proteins in the nuclear pore complex, which helps researchers to observe a clearer structure. This study provides new opportunities for studying the ultrastructural details of subcellular organelles or even biomacromolecules, using the conventional SMLM system.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation.

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

High-precision 3D drift correction with differential phase contrast images.

Single molecule localization microscopy (SMLM) usually requires long image acquisition time at the order of minutes and thus suffers from sample drift, which deteriorates image quality. A drift estimation method with high precision is typically used in SMLM, which can be further combined with a drift compensation device to enable active microscope stabilization. Among all the reported methods, the drift estimation method based on bright-field image correlation requires no extra sample preparation or complicated modification to the imaging setup. However, the performance of this method is limited by the contrast of bright-field images, especially for the structures without sufficient features. In this paper, we proposed to use differential phase contrast (DPC) microscopy to enhance the image contrast and presented a 3D drift correction method with higher precision and robustness. This DPC-based drift correction method is suitable even for biological samples without clear morphological features. We demonstrated that this method can achieve a correction precision of < 6 nm in both the lateral direction and axial direction. Using SMLM imaging of microtubules, we verified that this method provides a comparable drift estimation performance as redundant cross-correlation.

Two-color super-resolution localization microscopy via joint encoding of emitter location and color.

Multi-color super-resolution localization microscopy (SRLM) provides great opportunities for studying the structural and functional details of biological samples. However, current multi-color SRLM methods either suffer from medium to high crosstalk, or require a dedicated optical system and a complicated image analysis procedure. To address these problems, here we propose a completely different method to realize multi-color SRLM. This method is built upon a customized RGBW camera with a repeated pattern of filtered (Red, Green, Blue and Near-infrared) and unfiltered (White) pixels. With a new insight that RGBW camera is advantageous for color recognition instead of color reproduction, we developed a joint encoding scheme of emitter location and color. By combing this RGBW camera with the joint encoding scheme and a simple optical set-up, we demonstrated two-color SRLM with ∼20 nm resolution and < 2% crosstalk (which is comparable to the best-reported values). This study significantly reduces the complexity of two-color SRLM (and potentially multi-color SRLM), and thus offers good opportunities for general biomedical research laboratories to use multi-color SRLM, which is currently mastered only by well-trained researchers.

Computational framework for generating large panoramic super-resolution images from localization microscopy.

Combining super-resolution localization microscopy with pathology creates new opportunities for biomedical researches. This combination requires a suitable image mosaic method for generating a panoramic image from many overlapping super-resolution images. However, current image mosaic methods are not suitable for this purpose. Here we proposed a computational framework and developed an image mosaic method called NanoStitcher. We generated ground truth datasets and defined criteria to evaluate this computational framework. We used both simulated and experimental datasets to prove that NanoStitcher exhibits better performance than two representative image mosaic methods. This study is helpful for the mature of super-resolution digital pathology.

Divide and conquer: real-time maximum likelihood fitting of multiple emitters for super-resolution localization microscopy.

Multi-emitter localization has great potential for maximizing the imaging speed of super-resolution localization microscopy. However, the slow image analysis speed of reported multi-emitter localization algorithms limits their usage in mostly off-line image processing with small image size. Here we adopt the well-known divide and conquer strategy in computer science and present a fitting-based method called QC-STORM for fast multi-emitter localization. Using simulated and experimental data, we verify that QC-STORM is capable of providing real-time full image processing on raw images with 100 µm × 100 µm field of view and 10 ms exposure time, with comparable spatial resolution as the popular fitting-based ThunderSTORM and the up-to-date non-iterative WindSTORM. This study pushes the development and practical use of super-resolution localization microscopy in high-throughput or high-content imaging of cell-to-cell differences or discovering rare events in a large cell population.